Skip to content

Feature/sqlite compiletime#1180

Merged
Frotty merged 6 commits into
wurstscript:masterfrom
Code-Fixxers:feature/sqlite-compiletime
Jul 24, 2026
Merged

Feature/sqlite compiletime#1180
Frotty merged 6 commits into
wurstscript:masterfrom
Code-Fixxers:feature/sqlite-compiletime

Conversation

@Donach

@Donach Donach commented Apr 3, 2026

Copy link
Copy Markdown
Contributor

This PR adds compiletime support for sqlite operations. Not thoroughly tested, but simple stuff works:

Self-contained Examples:
SQLite.txt
SQLiteHelder.txt

@Frotty

Frotty commented Jun 6, 2026

Copy link
Copy Markdown
Member

Hi, as mentioned, will you do the standard library pendant? (provide api functions) else this isn't publicly usable unless users define @extern natives by themselves?

@Donach

Donach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Updated this PR with the clean implementation (including handle validation, statement connection ownership tracking, auto-finalizing open statements when closing connections, and explicit interpreter error messages).

Also opened the WurstStdlib2 PR providing the native @extern bindings: wurstscript/WurstStdlib2#450

@Frotty

Frotty commented Jul 22, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1742a4fbd7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Donach
Donach force-pushed the feature/sqlite-compiletime branch 3 times, most recently from d03161c to 5d16740 Compare July 22, 2026 22:23
@Donach
Donach force-pushed the feature/sqlite-compiletime branch from 5d16740 to 6e51eb3 Compare July 22, 2026 22:27
@Frotty

Frotty commented Jul 23, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e51eb3786

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Reborn Reviewer and others added 2 commits July 23, 2026 23:23
sqlite_reset previously called PreparedStatement.clearParameters(), wiping
all bound parameters. This diverges from SQLite's sqlite3_reset(), which
resets execution state but leaves bindings in place, and broke the common
reuse pattern of binding a parameter once, stepping, resetting, and stepping
again without rebinding.

sqlite_reset now only clears the execution state (open result set and the
executed marker). A separate sqlite_clear_bindings native maps to
sqlite3_clear_bindings() for callers that explicitly want to drop bindings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address adversarial review findings on the SQLite compiletime natives:

- sqlite_exec now runs every statement of a ';'-separated script instead of
  silently dropping all but the first (JDBC/xerial only executes the first).
  Splitting respects string literals, quoted identifiers and comments.
- sqlite_open rejects smuggled URI query parameters and explicitly disables
  extension loading, closing a load_extension() -> dlopen native-code vector
  reachable from a malicious compiletime dependency.
- sqlite_close now always closes the connection even if finalizing a
  statement throws, aggregating errors, so the connection can no longer leak.
- (re)binding a parameter invalidates prior execution state, so binding new
  values after a step (without an explicit sqlite_reset) re-executes instead
  of being silently ignored.
- Added sqlite_column_is_null so callers can distinguish SQL NULL from an
  empty string / zero value.
- closeAllSqliteResources no longer resets the handle counter, keeping
  handles monotonic so a stale handle can never alias a new resource.
- Documented the 1-based bind / 0-based column index convention and the
  32-bit truncation of sqlite_column_int.
- Removed dead, cleanup-escaping fallback branches in RunTests.

Tests: multi-statement exec, rebind-without-reset re-execution and NULL
detection compiletime tests, plus unit tests for the SQL statement splitter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Donach

Donach commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: binding preservation + hardening pass

Pushed two commits addressing the sqlite_reset review comment and an adversarial review of the whole SQLite native surface.

4bceaef92 — preserve bindings across sqlite_reset

sqlite_reset no longer calls PreparedStatement.clearParameters(). It now mirrors sqlite3_reset() — resets execution state (closes the open ResultSet, clears the executed marker) while leaving bound parameters in place, so the bind once → step → reset → step again reuse pattern works. Added sqlite_clear_bindings (maps to sqlite3_clear_bindings()) for callers that explicitly want to drop bindings.

4ba78432e — hardening pass

  • sqlite_exec ran only the first statement of a ;-separated script (JDBC/xerial silently drops the rest). Now splits and executes every statement, respecting string literals, quoted identifiers, and -- / /* */ comments.
  • sqlite_open extension-loading vector: rejects smuggled URI query params (e.g. ?enable_load_extension=true) and explicitly disables extension loading, closing a load_extension()dlopen native-code path reachable from a malicious compiletime dependency.
  • sqlite_close connection leak: the connection is now always closed even if finalizing a statement throws (errors are aggregated), instead of leaking a connection already removed from the handle map.
  • Rebind-without-reset silent no-op: (re)binding a parameter now invalidates prior execution state, so binding new values after a step re-executes on the next sqlite_step rather than being silently ignored.
  • NULL vs empty/zero: added sqlite_column_is_null (via ResultSet.wasNull()) so callers can distinguish a real SQL NULL.
  • Handle recycling: closeAllSqliteResources no longer resets the handle counter, keeping handles monotonic.
  • Documented the 1-based bind / 0-based column index convention and the 32-bit truncation of sqlite_column_int (inherent to WurstScript's 32-bit int).
  • Removed dead, cleanup-escaping fallback branches in RunTests.

Tests

  • Compiletime tests for reset-preserves-bindings, multi-statement exec, rebind-without-reset re-execution, and NULL detection.
  • Unit tests for the SQL statement splitter (semicolons inside string literals / quoted identifiers, comment stripping, empty-statement skipping).

All SQLite tests pass against sqlite-jdbc:3.46.1.3 on JDK 25.

Follow-up to the second adversarial review. Replace the hand-rolled SQL
statement splitter in sqlite_exec with a direct call to SQLite's native
sqlite3_exec (via the xerial driver's low-level DB handle). The splitter
could not parse trigger BEGIN...END bodies, CASE...END, or the [id] / `id`
identifier-quoting forms, so those scripts were shredded into invalid
fragments; delegating to SQLite's own parser handles all of them. The
splitter and its unit tests are removed; a native test now exercises a
multi-statement script containing a trigger and a ';'-bearing quoted
identifier.

Also from the review:
- sqlite_open now only rejects query parameters on the "file:" URI form,
  so a plain path legitimately containing '?' (e.g. on Linux) is accepted;
  enableLoadExtension(false) remains the defense-in-depth backstop for all
  paths (the extension-loading vector was verified closed either way).
- Documented that (re)binding a parameter closes any open result set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Frotty

Frotty commented Jul 24, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: deffe83656

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Add native-level tests proving each sqlite_* native's intended behavior and
its error contract, filling the gaps left by the happy-path-only tests:

- Round-trip of every column type (int/real/string) plus NULL -> "" and
  sqlite_column_is_null (both directions); sqlite_column_count before and
  after the first step.
- sqlite_step returns false past the last row.
- sqlite_reset rewinds a SELECT result set (re-reads from the first row).
- sqlite_column_int truncation of a 64-bit INTEGER (documented behavior).
- Native-level bind round-trip, rebind-after-step re-execution and
  binding persistence, and sqlite_clear_bindings resetting params to NULL.
- sqlite_finalize invalidates the statement handle; sqlite_close invalidates
  the connection and cascades to its statements.
- Invalid-handle and bad-SQL error paths across bind/step/reset/
  clear_bindings/finalize/column_*/prepare/exec/close.
- Security: sqlite_open rejects file: URIs with query parameters, and
  load_extension is not authorized on an opened connection.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Frotty

Frotty commented Jul 24, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cedda7b226

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

sqlite_clear_bindings called stmt.clearParameters() but, unlike the
sqlite_bind_* methods, did not markStatementForReexecution. So clearing
bindings after a statement had already been stepped left the executed
marker set (and any open result set cached): the next sqlite_step returned
false without re-running the statement, silently skipping execution with the
now-NULL parameters unless the caller also issued sqlite_reset.

Clearing bindings is a parameter mutation and now has the same
re-execution semantics as (re)binding.

Adds two regression tests covering both sub-cases the asymmetry produced —
the stale executed-marker (INSERT) path and the stale result-set (SELECT)
path — each verified to fail without this change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Donach

Donach commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Fixed: re-execute after sqlite_clear_bindings (8b9955b)

Confirmed and fixed. sqlite_clear_bindings called clearParameters() but — unlike the sqlite_bind_* methods — did not markStatementForReexecution. So after a statement had been stepped, clearing bindings left the executed marker set and any result set cached, and the next sqlite_step returned false without re-running, silently skipping execution with the now-NULL parameters unless the caller also called sqlite_reset.

Clearing bindings is a parameter mutation, so it now shares the exact re-execution semantics of (re)binding.

Regression tests (both verified to fail on the pre-fix code)

  • sqliteClearBindingsAfterStepReexecutesWithoutReset — stale executed-marker path (INSERT): clear-then-step now re-runs and inserts the NULL row.
  • sqliteClearBindingsAfterStepDiscardsOldResultSet — stale result-set path (SELECT): clear-then-step re-executes the query instead of continuing to walk the old cursor.

Swept the rest of the native surface for the same class of bug

The bug class is state-map inconsistency causing a silent no-op or stale read. I audited every state mutator against the sqliteExecutedStatements / sqliteResultSets invariants:

  • Parameter/execution mutatorssqlite_bind_int/real/string ✅ mark for re-execution; sqlite_clear_bindings ✅ now does; sqlite_reset ✅ closes the result set + drops the marker inline. That's the complete set — no other method mutates parameters or execution state, so this was the only remaining instance.
  • Handle allocation — connections and statements share one monotonic counter, so a statement handle can never alias a connection handle; finalized/closed handles are never reused. No aliasing loophole.
  • sqlite_finalize / sqlite_close — both remove the handle from every map (statements, connections, result sets, executed set); close finalizes child statements and always closes the connection even if a child finalize throws. No orphaned marker/result-set entries.
  • Reads past the last row / before a step — surface as loud InterpreterExceptions, not silent wrong results.

All 22 native unit tests pass, plus the full 354-test WurstStdlib2 suite (50 SQLite) compiled against this build.

@Frotty

Frotty commented Jul 24, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 8b9955bd17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Frotty
Frotty merged commit aec2342 into wurstscript:master Jul 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants